Learning Affordance for Direct Perception in Autonomous Driving

In the past decade, significant progress has been made in autonomous driving, and many novel computer vision methods have been proposed for vision-based autonomous driving. To date, most of these systems can be categorized into two major paradigms: mediated perception approaches and behavior reflex approaches. Mediated perception approaches [7] involve multiple sub-components for recognizing driving-relevant objects, such as lanes, traffic signs, traffic lights, cars, pedestrians, etc. [1]. The recognition results are then combined into a consistent world representation of the car’s immediate surroundings. To control the car, an AI-based engine will take all of this information into account before making each decision. Behavior reflex approaches construct a direct mapping from an image to a driving action. This idea dates back to the late 1980s when [5, 6] used a neural network to construct a direct mapping from an image to steering angles. We desire a representation that directly predicts the affordance for driving actions, instead of visually parsing the entire scene or blindly mapping an image to steering angles. In this paper, we propose a direct perception approach [3] for autonomous driving – a third paradigm that falls in between mediated perception and behavior reflex. We propose to learn a mapping from an image to several meaningful affordance indicators of the road situation, including the angle of the car relative to the road, the distance to the lane markings, and the distance to cars in the current and adjacent lanes. With this compact but meaningful affordance representation as perception output, we demonstrate that a very simple controller can then make driving decisions at a high level and drive the car smoothly. Our model is built upon the state-of-the-art deep Convolutional Neural Network (CNN) framework to automatically learn image features for estimating affordance related to autonomous driving. To build our training set, we ask a human driver to play a car racing video game TORCS [8] for 12 hours to collect screenshots and the corresponding ground truth values (from the game engine) of a few critical affordance indicators for driving, e.g. the host car’s relative position to the road’s central line, the distance to the preceding cars, etc. and train the CNN in a supervised learning learning manner. In the testing phase, at each time step, the trained model takes a driving scene image from the game and estimates the affordance indicators for driving. A driving controller processes the indicators and computes the steering and acceleration/brake commands. The driving commands are then sent back to the game to drive the host car (Figure 1).